Hand Tracking and Hand Gesture Recognition for Human Computer Interaction

Hand tracking and hand gesture recognition is an important problem in the field of human-computer interaction. A number of solutions have been proposed in the current literature, but the problem is still far from being solved since the hand exhibits significant amounts of articulation and self-occlusion that cause difficulties with existing algorithms. To further exasperate these problems, interactive applications require that the hand tracking perform in real-time. The current ubiquity of webcams offers an opportunity to create computer vision systems which can enable novel new methods for human-computer interaction. To that end, we present a system which allows the user to control the operating system cursor in a hands-free way by gesturing in mid-air. Our system leverages OpenCV and the X windowing system to track the index finger and thumb of a user using a webcam. The user can motion in the direction she/he wishes the cursor to move, and can execute mouse operations by “pointing” toward the camera and breaking a “plane of interaction”. Our goal with this work is to demonstrate a proof-of-concept system for enabling a new method of controlling and interfacing with a computer using commodity webcams.